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Transfer entropy provides a general tool for analyzing the magnitudes and directions — but not the kinds — of 
information transfer in a system. We extend transfer entropy in two complementary ways. First, we distinguish 
state-dependent from state-independent transfer, based on whether a source's influence depends on the state of 
the target. Second, for multiple sources, we distinguish between unique, redundant, and synergistic transfer. 
The new measures are demonstrated on several systems that extend examples from previous literature. 

PACS numbers: 89.70.Cf, 02.50.-r, 05.45.Tp, 89.75.-k 



I. INTRODUCTION 

Many scientific problems involve understanding the behav- 
ior of complex systems in terms of the interactions between 
their component parts. Using information theory, these inter- 
actions can be quantified as the information exchanged be- 
tween components HI. Such an approach has the advan- 
tages that informational measures are sensitive to arbitrary 
nonlinear interactions between components and have units of 
measurement (bits) that are easily interpreted and compared 
across systems. In particular, the information-theoretic mea- 
sure transfer entropy (TE) ||2] [51 has become widely adopted 
as a standard measure of information transfer, with applica- 
tions in neuroscience |^|, cellular biology L5J, chaotic syn- 
chronization ||6], and econophysics Q to name just a few. 

Transfer entropy provides a directional measure of the in- 
fluence that one random process, the source, has on another, 
the target. This influence is measured by the information that 
the source provides about the next state of the target when 
conditioned on the target's history. The idea behind TE is that 
conditioning on the target's history removes the information 
shared by the source and target due to common histories or 
inputs, thereby isolating the information that is actually trans- 
ferred. However, conditioning does not simply remove shared 
information; it also adds in higher-order synergistic informa- 
tion, an idea that was formalized in the recently proposed par- 
tial information (PI) decomposition ||8]. 

Here we apply this basic property of conditional informa- 
tion to generalize TE in two complementary ways. First, we 
decompose TE into two kinds of information transfer that dif- 
fer regarding the influence of the target's state. We show 
that the resulting measures are formally related to the control- 
theoretic concepts of open-loop and closed-loop control, and 
quantify separately the state-independent and state-dependent 
influences of the source onto the target. Second, we apply a 
similar decomposition to the case of multiple sources and de- 
rive a novel multivariate generalization of TE. The resulting 
measures quantify separately the unique, redundant, and syn- 
ergistic influences of multiple sources onto a target. Together 
these results provide a general framework for characterizing 
not only the magnitudes and directions but also the kinds of 
information exchange that occur between random processes. 



We begin by introducing Pl-decomposition for a system of 
three random variables, X, Y, and Z, for which we ask: How 
much total information do Y and Z provide about XI And, 
how do Y and Z contribute to the total information? The an- 
swer to the former is given by the mutual information (MI) 



I{X; Y, Z) = H{X) - H{X\Y, Z) 



(1) 



where H{X) — — J2xP(^) ^'^SPi^) is the familiar Shannon 
entropy [9 |. For the latter, we can identify three distinct pos- 
sibilities, that is, three kinds of information that Y and Z may 
provide. First, Y may provide information that Z does not, or 
vice versa (unique information). For example, if F is a copy 
of X and Z is a degenerate random variable, then the total in- 
formation reduces to the unique information from Y. Second, 
Y and Z may provide the same or overlapping information 
(redundancy). For example, if Y and Z are both copies of X 
then they redundantly provide complete information. Third, 
the combination of Y and Z may provide information that is 
not available from either alone (synergy). The classic example 
for binary variables is the exclusive-OR function X = Y (B Z, 
in which case Y and Z individually provide no information 
but together provide complete information. Thus, intuitively, 
I{X;Y, Z) decomposes into unique information from Y and 
Z, redundant information shared by Y and Z, and synergistic 
information contributed jointly by Y and Z. 

Pl-decomposition formalizes this idea, starting with a mea- 
sure of redundancy. Letting R = {Y, Z}, redundancy is de- 
fined as 



^ p{x) min /(X = x; R) (2) 



where 



I{X = x-R)=Y,p{r\x) 



log 



p{x) 



log 



p{x\r) 



(3) 



is the specific information that R provides about each state 
X — X. Thus, redundancy is defined as the minimum infor- 
mation that Y or: Z provides about each state of X, averaged 
over all possible states. This definition captures the idea that 
redundancy is the information shared by Y and Z (the min- 
imum that either provides) while taking into account that Y 
and Z may provide information about different states of X. 
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FIG. 1. Pl-decomposition of I{Xt+i; Xt,Yt) into unique informa- 
tion from Xt ({Xt}) and Yi ({Yt}), redundancy {{Xt}{Yt}), and 
synergy ({Xt, Vj}). Tlie gray region corresponds to Ty^x, whicli 
decomposes into SITE ({Yt}) and SDTE ({Xt, Yt}). 



Using /mill and the inclusion-exclusion principle lITOl . the 
total information I{X; Y, Z) can then be decomposed into 
partial information terms, given by the Pl-function IIr. The 
redundancy is given by Un{X; {Y}{Z}) = I^^X; Y, Z). 
The unique information from Y is given by li^iiX; {Y}) — 
I{X\Y)— /niin {X\ Y, Z), or the total information from Y mi- 
nus the redundancy, and likewise for Z. Finally, the synergy 
is given by UniX; {Y, Z}) = I{X- Y, Z) - I^,^{X; Y, Z), 
where /max is defined the same as /min except substituting 
max for min. Together these terms yield the decomposition: 

I{X; Y) = UniX; {Y}) + Ur{X; {Y}{Z}) (4) 
and 

I{X; r, Z) = Unix; {Y}) + Iln{X; {Z}) 

+ nR(X;{r}{z}) + nR(X;{r,z}). (5) 

An immediate consequence of Eqs. Q and (|5]l is that the 
conditional MI I{X; Y\Z) decomposes into the unique infor- 
mation from Y plus the synergy from Y and Z: 

I{X-Y\Z) = I{X-Y,Z) - I{X-Z) 

= n^{x-{Y}) + n^{x-{Y,z}). (6) 

Thus, conditioning I{X\ Y) on Z not only removes the re- 
dundancy from Y and Z, but also adds in their synergy. This 
observation makes intuitive sense if we think of I{X\ Y\Z) as 
answering the question: How much information do we gain 
from learning Y when we akeady know Zl Clearly, this will 
include both the information that comes uniquely from Y plus 
the synergistic information that comes from Y and Z together 
Transfer entropy can be analyzed in a similar way, since it 
is simply an application of conditional MI to stochastic pro- 
cesses. Given processes X and Y , the TE from y to X is 
defined as 

Ty^x =/(Xt+i;y/')|xf)) (7) 

(k) 

where XI ' is the fc-dimensional delay vector for X, and like- 
wise for y/'^ and Y (henceforth the superscripts are omit- 
ted for clarity). In other words, Ty^x quantifies the in- 
formation that previous values of Y provide about the next 
state of X when conditioned on X's own history. In terms 



of transition probabilities, Ty^x can also be thought of as 
quantifying deviation from the generalized Markov property 
v{'^t^\Vt,Vt) v{'^t^iVt), with Ty^x = iff y has no 
influence on the transitions of X. 



11. DECOMPOSING TRANSFER ENTROPY 

Our first main result is that, by decomposing Ty^x, we can 
distinguish two kinds of information transfer; that is, two dis- 
tinct ways that Y can influence the transitions of X (FIG.[T]l. 
Letting R = {Xt,Yt} and combining Eqs. (|6]l and (|7]i, we 
have that 

Ty^x ^TlniXt+uiYt}) +Un{X,+i;{Xt,Yt}) (8) 

where ^niXt^i; {Yt}) is the unique information that Yt 
provides about Xt+i and Il^{Xt+i; {Xt,Yt}) is the syn- 
ergistic information from Xt and Yt. As we will show, 
^iiiXt+i', {Yt}) corresponds to state-independent transfer 
entropy (SITE): it measures the portion of y^'s influence on 
Xt+i that does not depend on Xt- The complementary term 
^niXt+i, {Xt,Yt}) is the state-dependent transfer entropy 
(SDTE): it measures the influence that Yt has on Xt+i only 
when combined with an appropriate state of Xt- To ground 
this interpretation, we next establish a formal connection be- 
tween SITE and SDTE and the control-theoretic notions of 
open-loop and closed-loop control. 

In control theory, one considers a process Xt — 
characterized by its initial state X and final state X' — and a 
controller C, with the two related by a distribution p{x'\x, c) 
1 1 1 , 12 1 . The aim is to specify a control policy, given by the 
distribution p{c\x), that moves the system to certain desired fi- 
nal states. In open-loop control, the controller C acts indepen- 
dently of the initial state X (I{X; C) = 0), while closed-loop 
control is characterized by state-dependent actuation. 

A fundamental property of a control system is its control- 
lability, which is the extent to which it can be moved through 
its entire state space. In particular, a system has perfect con- 
trollability iff there is a control policy that moves the system 
deterministically from any x E X to any x' £ X' . In lfT2ll . it 
is shown that a natural information-theoretic measure of con- 
trollability is I{X'; C\X) — the information transfer from the 
controller to the controlled process — which is maximal ex- 
actly in the case of perfect controllability. Thus, there is a 
close parallel between information transfer and controllabil- 
ity, where essentially the only difference is semantic: infor- 
mation transfer applies to arbitrary interactions between pro- 
cesses, while controllability is concerned specifically with us- 
ing one process to influence another 

With this in mind, the following result connects SITE and 
SDTE with open-loop and closed-loop control (proof in Ap- 
pendix |A]). 

Theorem 1. A system is perfectly controllable with open- 
loop control iff it is perfectly controllable with only state- 
independent transfer from C to X' . 

Thus, decomposing I(X' ■,C\X) as in Equation (|8]l, SITE 
from C to X' measures a system's open-loop controllabil- 
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FIG. 2. SITE (black) and SDTE (gray) for binary Markov processes 
X and y as a function of the coupling parameter d. 



ity (maximal for perfect open-loop control), while SDTE 
measures the additional contribution from closed-loop con- 
trol. More generally, this connection grounds the interpre- 
tation of SITE as the state-independent (open-loop) influence 
of one process on another, and likewise for SDTE and state- 
dependent (closed-loop) influence. 

As a simple example to illustrate the two kinds of trans- 
fer, consider two binary state Markov processes X and Y, 
where Y is purely random and X is stochastically coupled to 
Y. Specifically, if xt — 0, then Xf+i = yt, while if xt = 1, 
the probability that Xt+i = is 1 — d and that Xt+i = 1 — yt 
is d. A simple eigenvector calculation yields the stationary 
distribution p{x, y) = 1/4 for all x and y, and from this all in- 
formational quantities can be computed. When d — 0, Xt+i 
is simply set to Yt regardless of its own previous state, thus 
corresponding to pure SITE (FIG. |2]i. With this parameter 
setting, the system is essentially equivalent to the discrete ex- 
ample considered in ||3J. In contrast, when d = 1, yi = 
causes X to remain in the same state and yt = I causes X to 
switch states. Consequently, Y's influence on Xt+i depends 
entirely on Xt, corresponding to pure SDTE. In fact, if one 
imagines using Y to control X, then d = 1 corresponds to 
a 'controlled-NOT' gate, which is known to require closed- 
loop control 111 llll2il . FIG.|2]shows how varying d produces a 
smooth transition between these two extremes. 

Finally, we note that the distinction between SITE and 
SDTE also clarifies the relationship between TE and the time- 
delayed mutual information (TDMI) I{Xt+i; Yt), which was 
the standard measure of information transfer prior to TE |2|. 
Transfer entropy was initially proposed as an alternative to 
TDMI because the latter fails to remove shared information 
due to common histories or inputs. From FIG.[T] it is clear that 
this shared information corresponds to Il^{Xt+i; {Xt}{Yt}), 
the redundancy between Xt and Yt. However, FIG. [T] also 
reveals a second crucial difference between TDMI and TE, 
which is that TDMI fails to include SDTE. Thus, not only 
does TDMI incorrectly add in shared information, but it also 
leaves out a significant component of information transfer. 



III. MULTIVARIATE INFORMATION TRANSFER 

Our second main result is a novel multivariate generaliza- 
tion of TE, based on applying Pl-decomposition to the infor- 
mation from multiple sources. Schreiber L2J originally pro- 



posed a generalization of TE based on 'conditioning out' other 
sources, an idea that has since been adopted and extended 
by others |fT3l . However, it should be clear from the preced- 
ing discussion that such a generalization is problematic, since 
conditioning does not simply remove shared information. Our 
generalization addresses this deficiency by quantifying sep- 
arately the unique, redundant, and synergistic transfer from 
multiple sources. 

For simplicity, we consider only two sources Y and Z 
acting on a target X (the general case is discussed momen- 
tarily), in which case the total TE is given by Ty^z^x = 
I{Xt+i;Yt, Zt\Xt). Applying Pl-decomposition as before, 
we arrive at measures for the redundant transfer from Y and 
Z'- Tf^Y}{z}^x = IininiXt+i;Yt, ZtlXt); the unique trans- 
fer from Y (resp. Z): Ty^x\z = Ty^x - T{y}{z}^x; 
and the synergistic transfer from Y and Z: T^y.zj^x — 
Tyz^x - ^max(^t+i;i^i,2't|Xt). Generally speaking, re- 
dundant transfer corresponds to situations where the apparent 
influence from multiple sources may in fact be due to any one 
(or several) of them, indicating that interventional methods 
are required to determine the true causal structure |T4|. In 
contrast, unique transfer represents the portion of a source's 
influence that can only come from that source, or, if all possi- 
ble sources are considered, that must come from that source. 
Finally, synergistic transfer indicates that several sources act 
together cooperatively to influence the target. 

To illustrate, consider three binary state Markov processes 
X, Y, and Z. Y is purely random, and Z is stochastically 
coupled to Y such that zt = yt with probability (1 + c)/2 
and Zt = I — yt with probability (1 — c)/2. This coupling 
can be thought of as an external signal driving Y and Z to 
synchronize: as c goes from to 1, F and Z transition from 
independence to complete synchronization. X in turn is cou- 
pled to both Y and Z such that, if zt = 0, Xt+i — yt, 
while if Zt = 1, xt+i — yt with probability (1 — d) and 
xt+i = (1 — yt) with probability d. Thus, Xf+i = yt when 
d = 0, and xt+i = yt® Zt when d = 1. At the extreme pa- 
rameter settings, this system exhibits three different behaviors 
(FIG.[3]|. With (c = 0, d = 0), Y and Z are independent and 
X depends only on Y , so the only influence is unique transfer 
from Y to X. In contrast, with (c = l,d — 0), X again de- 
pends only on Y but Y and Z are now synchronized, so there 
is only redundant transfer from Y and Z. Indeed, in this case 
it is impossible to determine from observation alone whether 
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FIG. 3. Ty^x\z (blue), Tiy}{z)^x (green), and Tiy^z}~>x (red) 
for binary Markov processes X, Y, and Z. 
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FIG. 4. Information transfer I{Bt+i; Ht\Bt = &t) from heart rate 
(H) to breath rate (B) as a function of bt for bandwidth r — 0.5. 
Quahtatively similar results were found for r G [0.2, 1.0]. 



Y 01 Z (or both) is driving X. Finally, with (c = l,d = 0), 

Y and Z are independent and Xt+i —Yt® Zt, corresponding 
to pure synergistic transfer. 

As a final example, we extend the analysis of a multivariate 
physiological time series presented in ||2] E]. The data con- 
sists of simultaneous recordings of the breath rate (chest vol- 
ume), heart rate, and blood oxygen concentration for a patient 
suffering from sleep apnea. Previous analysis compared TE 
and TDMI for both directions between the breath and heart 
signals. However, directly comparing TE and TDMI is prob- 
lematic and potentially misleading, since both measures de- 
tect SITE but differ regarding SDTE and shared information 
(FIG.[T]|. Indeed, with no additional information, it is impos- 
sible to determine even whether TE and TDMI are detecting 
the same or different aspects of an interaction. 

To address this issue, we calculated SITE and SDTE be- 
tween the breath and heart signals. Joint probability estimates 
were obtained by kernel estimation using a rectangular ker- 
nel with bandwidth r. Neighboring points closer than 20 time 
steps were excluded and points with fewer than 5 neighbors 
were ignored, following the suggestions of Schreiber and oth- 
ers 1 2 3 5 |. Our main finding is that SITE is consistent with 
zero in both directions between the breath and heart signals. 
Thus, for these signals, TE and TDMI in fact quantify entirely 
separate things: TDMI is due only to shared information from 
common histories or inputs, while TE detects state-dependent 
information exchange. This state dependence can be seen by 
plotting information transfer as a function of the target state, 
shown in FIG|4]for T(heart ^ breath). For pure SITE, this 
plot would be uniform across target states, while FIG.|4]shows 
a clear bimodal distribution. These two modes coiTespond to 
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FIG. 5. To^B\H (black), Th^b\o (gray), T{h}{0}-^b (dashed 
black), and 7{h.o}^s (dashed gray) from heart rate (H) and blood 
oxygen (O) to breath rate (B). 



downswings and upswings in chest volume, suggesting that 
heart rate has the largest influence on respiration when chest 
volume is low and, to a lesser extent, when it is high, but min- 
imal influence when chest volume is near its mean (see also 
FIG.|6]in Appendix|B]). 

We also analyzed the combined influence of heart rate 
and blood oxygen level on breath rate (FIG. [5]l. The most 
significant component is consistently the unique information 
transfer from the heart rate, indicating that most of the TE 
discussed above is uniquely attributable to the heart signal. 
However, there is also considerable redundant and synergis- 
tic transfer, of roughly comparable magnitude, from heart rate 
and blood oxygen concentration. In contrast, there is essen- 
tially no unique information transfer from blood oxygen con- 
centration, indicating that all of its apparent influence could 
also be due to heart rate. 

We conclude by noting that the multivariate generalization 
described here extends naturally to any number of sources, 
simply by applying the general form of Pl-decomposition 1 8 1 . 
The two extensions described in this Letter can also be applied 
in conjunction, allowing one to quantify, e.g., state-dependent 
synergistic transfer. Thus, together these methods provide a 
completely general framework for characterizing information 
exchange in complex systems. 



Appendix A: Proof of Theorem 1 

As defined in fV2\, a system has perfect controllability iff 
for any initial state x and final state x' there exists a controller 
state c such that p(x'\x, c) — 1. We first prove that an equiv- 
alent definition of perfect controllability is that a system can 
be moved deterministically to any final state from any distri- 
bution of initial states. 

Lemma 1. A system is perfectly controllable iff for any x' 
there exists a distribution p{c\x) such that p{x') = Ifor any 
distribution p{x). 

Proof. If a system is perfectly controllable, we know that for 
a given x' there exists at least one c for each x such that 
p{x'\x, c) — 1. Thus, we can choose 



supp(C|a;) = {c : p{x'\x, c) = 1} 



(Al) 



for each x E X, which guarantees that p{x') — 1 for any 
distribution p{x). As this is verified for any x', this proves the 
direct part of the theorem. 

Conversely, note that if p{x') = 1 for a given x' and any 
distribution p{x), it must be that 

p{x'\x) = ^p(x'|x,c)p(c|a:) = 1 (A2) 

c 

for each x G X, and thus for each x there must be at least 
one c for which p{x'\x,c) = 1. As this holds for any x', the 
converse is proven. □ 

In order for a system to be perfectly controllable via open- 
loop control, we also require that the controller acts indepen- 
dently of the initial state, leading to the following definition. 
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Definition 1. A system has perfect open-loop controllabil- 
ity iff for any x' there exists a distribution p{c\x) such that 
p{x') = Ifor any distribution p{x), and I{X; C) — 0. 

An alternative definition of perfect open-loop controllabil- 
ity is given by the following lemma. 

Lemma 2. A system has perfect open-loop controllability iff 
for any x' there exists a c such that p{x'\c) — 1. 

Proof If I{X; C) = 0, then p{x') can be written as 

P{x') = ^P{c) ^p{x)p{x'\x, c). (A3) 

C X 

If in addition we have that p(x') = 1 for a given x' and 
any distribution p{x), then there must exist a c for which 
p{x'\xtC) = 1 for all x, i.e., p{x'\c) = 1. This holds for 
any x', so the direct part of the theorem is proven. 

Conversely, if for any x' there exists a c such thatp(a;'|c) = 
1, then for a given x' we can choose 



Proof. 



supp(C) = {c:p(x'|c) = l} 



(A4) 



with p{c) = p{c\x) for all c and x, ensuring that p{x') — 1 
and I{X; C) = 0. As this holds for any x', the converse is 
proven. □ 

In lfT2l . it is shown that an equivalent information-theoretic 
definition of perfect controllability is that there exists a distri- 
bution p{c\x) such that each final state is reachable from each 
initial state, i.e.. 



p{x'\x) ^ 



(A5) 



for all X and x' and that, for any distribution I{X'; C\X) 
is maximal, i.e.. 



H{X'\X,C)^0 



(A6) 



so that 



IiX'-C\X) = H{X'\X) - H{X'\X, C) = H{X'\X). 

(A7) 

Consequently, I{X';C\X) is naturally interpreted as a sys- 
tem's degree of controllability, which is maximal iff the sys- 
tem is perfectly controllable. 

By the same reasoning. Theorem 1 establishes that the SITE 
from C to X' is a natural measure of a system's open-loop 
controllability, maximal exactly in the case of perfect open- 
loop control. In order to prove Theorem 1, we will need the 
following basic property of SDTE. 

Lemma 3. The SDTE from C to X' is zero iff for each x' e 
X', 

p{x'\x, c) — p{x'\x) Vx, c 



Ii^{X'-{X,C}) 

=I{X';X, C)-Im.AX';X, C) 

= Y.p{x')[l{X' ^x';X,Cy 

x' 

max{/(X' = x']X),I{X' = x'\C)} 
= ^p{x') min{I{X' = x'\ C\X),I{X' = x' ■ X\C)} 

x' 

where 

I{X' = x'-C\X) = ^p(a;|x')^(p(c|a.,x') |1 p{c\x)) 

X 

and £)(• || •) is the KuUback-Leibler divergence |9|. D{- \\ •) 
is nonnegative, so nR(X'; {X, C}) = iff, for each x' £ X' , 
I{X' = x';C\X) = Oor/(X' = x';X\C) = 0. Further- 
more, since D{q || r) = iff q = r, Ur{X';{X,C}) = 
iff, for each x' e X\ 

p{c\x, x') = p{c\x) Vx, c 
or 



or, equivalently. 



p(a:|c, x') — p{x\c) \fx, c 



p{x'\x, c) = p{x'\x) Vx, c 
or 

p{x'\x, c) — p{x'\c) Vx, c. 



Now we are in a position to prove Theorem 1 . 



□ 



Proof We will show that a system has perfect open-loop con- 
trollability iff there exists a distribution p{c\x) such that Eqs. 
( |A5| l and ( |A6| l are satisfied and there is no SDTE from C to 

X', 



I1r{X';{X, C})^Q. 



(A8) 



If a system is open-loop controllable, then for each x' there 
exists a c such that p{x'\c) = 1. Choosing 



supp(C) = {c:p(x'|c) = l} 



(A9) 



over all x' E X', with p{c) = p{c\x) for all c and x, en- 
sures that H{X'\X,C) < H{X'\C) = and p{x'\x) 7^ 0. 
Also, the chosen distribution p{c\x) ensures that p{x'\x, c) = 
p{x'\c) ^x, c so that Il-[i{X': {X, C'}) = by the preceding 
lemma. This proves the direct part of the theorem. 

For the converse, note that p{x'\x) 7^ for a given x' and 
X means that there is at least one c for which p{x'\x, c) ^ 
and, since H{X'\X,C) — 0, we can further conclude that 
pix'\x,c) = 1. If we also have that nR(X';{X,C}) = 0, we 
know that, for each x' e X', 

p{x'\x, c) — p{x'\x) Vx, c 



p{x'\x, c) — p{x'\c) Vx, c. 



p(x'|x, c) = p{x'\c) Vx, c 
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FIG. 6. Information transfer I{Bt+i; Ht\Bt = 6t) from heart rate (H) to breath rate (B) superimposed on the breath rate signal. Darker colors 
correspond to higher values of I{Bt+i; Ht\Bt — bt). Heart rate has the largest influence on breath rate when chest volume is low and, to a 
lesser extent, when it is high, but minimal influence when chest volume is near its mean. 



and thus that, for each x' E X', 

3x,p{x'\x) = 1 
or 

3c,p{x'\c) = 1. 

But it cannot be the case that 3x,p{x'\x) = 1, since that 
would violate the reachability condition (Eq. ( |A5| l), so we 
conclude that for each x' there exists a c such that p{x'\c) — 1. 
This proves the converse. □ 



This is shown most clearly by superimposing the information 
transfer values on the breath rate signal (FIG.|6]l. 
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